Support `eventstats` command with Calcite by LantaoJin · Pull Request #3585 · opensearch-project/sql

LantaoJin · 2025-04-27T09:58:54Z

Description

Support eventstats command with Calcite

The eventstats command actually equals to window function in SQL

source=t | eventstats avg(a), count() by b

equals to

SELECT avg(a) OVER (PARTITION BY b), count(*) OVER (PARTITION BY b) FROM t

And the default window frame is ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

In this PR, the supported window functions are

max
min
sum
count
avg
var_samp
var_pop
stddev_samp
stddev_pop

We will support more window functions in follow-up PRs since they are not supported in PPL-Spark

   : ROW_NUMBER
   | RANK
   | DENSE_RANK
   | PERCENT_RANK
   | CUME_DIST
   | FIRST
   | LAST
   | NTH
   | NTILE

Related Issues

Resolves #3563

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

xinyual · 2025-04-28T09:28:21Z

core/src/main/java/org/opensearch/sql/ast/tree/Window.java


-  private final UnresolvedExpression windowFunction;
-
+  private final List<UnresolvedExpression> windowFunctionList;


do we need to add window boundary? For example, I want to execute a ppl equal to a sql like
select AVG(SUM(total_amount)) OVER (ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS moving_avg_3day (this is one of the moving avg, which we plan to support for t2visbuilder).

It's easy to support it. but not this PR of eventstats command. We can extend this syntax after PPL lang unified. e.g support boundary when we implement streamstats command.

I have added a WindowFrame for further using.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

penghuo · 2025-04-28T18:37:20Z

docs/user/ppl/cmd/eventstats.rst

+ * ``eventstats``: Useful when you need to enrich events with statistical context for further analysis or filtering. Can be used mid-search to add statistics that can be used in subsequent commands.
+
+
+Version


nice!
Could we highlight version info? for instance, in title, evenstats (since 3.1.0). or maybe using version controled doc? Any thoughts? @dai-chen @qianheng-aws

highlighted in description and docs/user/ppl/index.rst

penghuo · 2025-04-28T18:40:24Z

docs/user/ppl/cmd/eventstats.rst

+
+Example::
+
+    PPL> source=accounts | eventstats sum(age) by gender;


add an example explain how to handle missing value?

penghuo · 2025-04-28T19:54:49Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

+      if (BuiltinFunctionName.AVG == agg.get()) {
+
+      } else {
+      }


penghuo · 2025-04-28T20:03:40Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLEventstatsIT.java

+        rows("John", "Canada", "Ontario", 4, 2023, 25, 4, 36.25, 20, 70),
+        rows("Jake", "USA", "California", 4, 2023, 70, 4, 36.25, 20, 70),
+        rows("Jane", "Canada", "Quebec", 4, 2023, 20, 4, 36.25, 20, 70),
+        rows("Hello", "USA", "New York", 4, 2023, 30, 4, 36.25, 20, 70));


in scheam, 2nd column is age, but in result, 2nd column is country?

The data order is the actually schema order. Let me change to verifySchemaInOrder

Signed-off-by: Lantao Jin <ltjin@amazon.com>

penghuo · 2025-05-02T15:10:02Z

@dai-chen please take another look.

qianheng-aws · 2025-05-06T05:05:12Z

core/src/main/java/org/opensearch/sql/calcite/utils/PlanUtils.java

+
+  static RexNode makeOver(
+      CalcitePlanContext context,
+      String name,


[minor] Why not just pass in BuiltinFunctionName here?

qianheng-aws · 2025-05-06T05:09:45Z

core/src/main/java/org/opensearch/sql/calcite/utils/PlanUtils.java

+    RexWindowBound lowerBound = convert(context, windowFrame.getLower());
+    RexWindowBound upperBound = convert(context, windowFrame.getUpper());
+    switch (functionName) {
+        // There is no "avg" AggImplementor in Calcite, we have to change avg window


There is a Calcite rule AggregateReduceFunctionsRule, it will do such transformation of rewriting avg to sum/count. Could we just use avg here and leverage that rule for transformation? It should introduce more refined handling for cases like NULL values.

Thanks for informing this, let me check how to apply this rule.

After checking AggregateReduceFunctionsRule code, seems it could be applied by logical Aggregate, rather than logical Window. The converting made by SqlNode level (parser). We can open a follow-up ticket about self-dev rule for window.

qianheng-aws · 2025-05-06T05:15:35Z

core/src/main/java/org/opensearch/sql/calcite/CalcitePlanContext.java

    }
  }

+  public void pushWindowPartitions(List<RexNode> partition) {


Seems never used for these 3 APIs. And why should the window related context stored in this global context?

Will delete them. It is useless code.

qianheng-aws · 2025-05-06T05:50:13Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+                    .map(
+                        groupCtx ->
+                            (UnresolvedExpression)
+                                new Alias(


[question] Why do we always need to wrap byClause with Alias here?

could be a followup and it needs much code refactor for stats, not sure the original reason.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* Support eventstats command with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * add doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * Support variance functions Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix UT Signed-off-by: Lantao Jin <ltjin@amazon.com> * update doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>

Support eventstats command with Calcite

c017a1c

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin added PPL Piped processing language calcite calcite migration releated labels Apr 27, 2025

add doc

6a59baf

Signed-off-by: Lantao Jin <ltjin@amazon.com>

xinyual reviewed Apr 28, 2025

View reviewed changes

fix IT

23c7dfa

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin marked this pull request as ready for review April 28, 2025 11:43

LantaoJin requested review from GumpacG, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, qianheng-aws, seankao-az and ykmr1224 as code owners April 28, 2025 11:43

dai-chen reviewed Apr 28, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java Outdated Show resolved Hide resolved

penghuo reviewed Apr 28, 2025

View reviewed changes

LantaoJin added 2 commits April 29, 2025 17:18

address comments

85127d8

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Merge remote-tracking branch 'upstream/main' into pr/issues/3563

5a4c3cc

Fix conflicts

13d5b0c

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin requested review from dai-chen, penghuo and xinyual May 1, 2025 13:21

penghuo previously approved these changes May 2, 2025

View reviewed changes

qianheng-aws reviewed May 6, 2025

View reviewed changes

address comments

65acffc

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin dismissed penghuo’s stale review via 65acffc May 6, 2025 07:56

Support variance functions

f54cf56

Signed-off-by: Lantao Jin <ltjin@amazon.com>

qianheng-aws previously approved these changes May 6, 2025

View reviewed changes

fix UT

843835b

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin dismissed qianheng-aws’s stale review via 843835b May 6, 2025 13:04

update doc

44f2455

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin requested review from penghuo and qianheng-aws May 6, 2025 13:17

penghuo approved these changes May 6, 2025

View reviewed changes

qianheng-aws approved these changes May 7, 2025

View reviewed changes

LantaoJin merged commit e51fcd9 into opensearch-project:main May 7, 2025
22 checks passed

LantaoJin mentioned this pull request Jun 16, 2025

[FEATURE]Add ppl eventstats command functionality #3024

Closed

dai-chen mentioned this pull request Aug 15, 2025

[FEATURE] PPL eventstats command enhancement #4047

Open


		private final UnresolvedExpression windowFunction;

		private final List<UnresolvedExpression> windowFunctionList;

		* ``eventstats``: Useful when you need to enrich events with statistical context for further analysis or filtering. Can be used mid-search to add statistics that can be used in subsequent commands.


		Version


		Example::

		PPL> source=accounts \| eventstats sum(age) by gender;

Conversation

LantaoJin commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo commented May 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LantaoJin commented Apr 27, 2025 •

edited

Loading

LantaoJin Apr 28, 2025 •

edited

Loading

LantaoJin Apr 29, 2025 •

edited

Loading

LantaoJin May 6, 2025 •

edited

Loading